Field of the Invention
The present invention relates to an information processing apparatus, an information processing method, and a computer-readable storage medium.
Description of the Related Art
In recent years, MR (Mixed Reality) technology is a subject of active study. MR technology refers to a technology of seamlessly combining a real space with a virtual space generated by a computer. MR technology is expected to be applied to various fields, such as assembly support, for displaying the operation procedure and the state of wiring in superposition at the time of an assembly operation, and surgery support, for displaying the state of the interior of the body of a patient in superposition on his or her body surface.
To make the user feel as if a virtual object really exists in a real space, geometrical consistency between the virtual object and the real space is of prime importance. The geometrical consistency in MR (Mixed Reality) includes consistency for matching the coordinate system of a real space with that of a virtual space, and consistency for correctly representing the depth relationship between a real object and a virtual object. A problem associated with the former consistency is also called a registration problem in MR (Mixed Reality), and various research reports have been presented (see, for example, K.Satoh, S.Uchiyama, and H.Tamura, “Registration Techniques in Mixed Reality,” Transactions of the Virtual Reality Society of Japan, Vol. 8, No. 2, pp. 171-180, 2003). The registration problem is posed when the position and orientation are matched between an image capturing apparatus for capturing a video at the observer's eyepoint, and a real object on which virtual information is to be displayed in superposition. This registration problem will be dealt with in the present invention.
As methods of measuring the position and orientation of an image capturing apparatus,
(1) a method of using the measurement values obtained by a sensor capable of simultaneously measuring both the position and orientation,
(2) a method of using the measurement value obtained by a sensor that measures the orientation, and image information of an index detected from an image captured by the image capturing apparatus,
(3) a method of using image information of an index detected from an image captured by the image capturing apparatus to correct the measurement values obtained by a sensor capable of simultaneously measuring both the position and orientation, are typically available (see, for example, K.Satoh, S.Uchiyama, and H.Tamura, “Registration Techniques in Mixed Reality,” Transactions of the Virtual Reality Society of Japan, Vol. 8, No. 2, pp. 171-180, 2003, and K.Satoh, S.Uchiyama, and H.Yamamoto, “A head tracking method using bird's-eye view camera and gyroscope,” Proc. 3rd IEEE/ACM Int'l Symp. on Mixed and Augmented Reality (ISMAR 2004), pp. 202-211, 2004). In general, a magnetic or an optical sensor is used as a position and orientation sensor that measures the position and orientation, while a gyro inertial sensor is used as an orientation sensor that measures the orientation.
To mount an orientation sensor or a position and orientation sensor on an image capturing apparatus to obtain the position or orientation of the image capturing apparatus or object using the sensor measurement value, it is necessary to measure, in advance, the position or orientation of the orientation sensor or the position and orientation sensor relative to the image capturing apparatus to be measured. Obtaining the arrangement information, that is, the position or orientation of the sensor relative to the object to be measured will be referred to as “sensor calibration” hereafter.
Japanese Patent Laid-Open No. 2006-153848 discloses a sensor calibration method of mounting an index and a position and orientation sensor, or an orientation sensor, on an object to obtain, as calibration information, the arrangement information of the position and orientation sensor or the orientation sensor mounted on the object relative to the object. An image is obtained by capturing an object, a sensor measurement value is input, and information associated with the image coordinate position of an index is detected from the image, thereby obtaining calibration information using the sensor measurement value and the information associated with the image coordinate position of the index. The measurement value obtained by a position and orientation sensor or an orientation sensor, and the image information of an index detected in an image, which are required to calibrate the sensor using the image, will be referred to as calibration input data hereafter.
To calibrate the sensor, it is necessary to obtain calibration input data at various positions and orientations. Also, because the accuracy of calibration input data considerably influences the sensor calibration result, it is necessary not only to input the measurement values obtained by a highly accurate sensor, but also to detect an index from a captured image with high accuracy. When the index is captured as a small object in the image, or a blur occurs in a captured image as the image capturing apparatus rotates or moves at a high speed, the index detection accuracy degrades. Therefore, the accuracy of the calibration result degrades when sensor calibration input data obtained in such a case is input and used for calculation.
Also, in Japanese Patent Laid-Open No. 2008-046687, to reduce calibration input data that leads to degradation in accuracy of sensor calibration so as to improve the calibration accuracy, a virtual three-dimensional object that indicates a target image capturing position and orientation is presented in obtaining calibration input data.
As in Japanese Patent Laid-Open No. 2006-153848, however, when the sensor is calibrated based on the sensor measurement value and the image information of an index, it is difficult for an operator who has neither knowledge nor expertise in sensor calibration to determine whether the input data is suitable for this calibration.
This is because, even if the measurement values fall within the measurement area of the position and orientation sensor, they do not always have a uniform measurement accuracy, and locally vary in sensor measurement accuracy, depending on the surrounding environment and the measurement position. More specifically, a magnetic position and orientation sensor is known to be vulnerable to a surrounding magnetic field and incapable of obtaining correct measurement values, depending on the magnetic field strain. Also, an optical position and orientation sensor has a position or orientation measurement accuracy, which locally varies depending on the accuracy of its lens distortion correction parameters and camera parameters. When this happens, the measurement accuracy is often high in the vicinity of the lens center; that is, at the center of the measurement area of the optical position and orientation sensor, but low in the vicinity of, for example, the boundary of the measurement area. Therefore, for accurate sensor calibration, the operator must obtain calibration input data in consideration of the characteristics and surrounding environment of the sensor.
It is also necessary to capture the index as a sufficiently large object in the image at various positions and orientations with no blur within the measurement area of the sensor. Especially, when the sensor measurement area is wide, it is difficult for an operator who has no expertise to obtain the sensor measurement values at various positions and orientations while determining whether the data is suitable for calibration.
In Japanese Patent Laid-Open No. 2008-046687, the target position and orientation of the image capturing apparatus can be presented to obtain calibration input data. However, this patent literature discloses no technique of determining whether the calibration input data is suitable at the position where calibration input data is obtained, in consideration of the characteristics and surrounding environment of the sensor. In addition, to present a correct target position and orientation, it is necessary to obtain an accurate position and orientation of the image capturing apparatus. However, since sensor calibration is done to obtain an accurate position and orientation of the image capturing apparatus, it is naturally impossible to obtain an accurate position and orientation of the image capturing apparatus before sensor calibration.
Hence, although a target position and orientation can be presented based on an approximate calibration result, it is difficult to determine whether the input data is suitable for calibration at the position where calibration input data is actually obtained.
The present invention has been made in consideration of the above-mentioned problem, and provides a technique of allowing even an operator who has neither knowledge nor expertise in sensor calibration to easily obtain sensor calibration input data, and to determine whether the calibration input data is suitable for accurate calibration.